Feature extraction and machine learning for evaluation of image- or video-type, media-rich coursework

ABSTRACT

Conventional techniques for automatically evaluating and grading assignments are generally ill-suited to evaluation of coursework submitted in media-rich form. For courses whose subject includes programming, signal processing or other functionally expressed designs that operate on, or are used to produce media content, conventional techniques are also ill-suited. It has been discovered that media-rich, indeed even expressive, content can be accommodated as, or as derivatives of, coursework submissions using feature extraction and machine learning techniques. Accordingly, in on-line course offerings, even large numbers of students and student submissions may be accommodated in a scalable and uniform grading or scoring scheme. Instructors or curriculum designers may adaptively refine assignments or testing based on classifier feedback. Using developed techniques, it is possible to administer courses and automatically grade submitted work that takes the form of media encodings of artistic expression, computer programming and even signal processing to be applied to media content.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation-in-part of U.S. applicationSer. No. 14/448,579, filed Jul. 31, 2014, which (in turn) claimspriority under 35 U.S.C. §119(e) of U.S. Provisional Application No.61/860,375, filed Jul. 31, 2013. Each of the foregoing applications isincorporated herein by reference.

BACKGROUND

Field of the Invention

The present application is related to automated techniques forevaluating work product and, in particular, to techniques that employfeature extraction and machine learning to efficiently and consistentlyevaluate instances of media content that constitute, or are derivedfrom, coursework submissions.

Description of the Related Art

As educational institutions seek to serve a broader range of studentsand student situations, on-line courses have become an increasinglyimportant offering. Indeed, numerous instances of an increasinglypopular genre of on-line courses, known as Massive Open Online Courses(MOOCs), are being created and offered by many universities, as diverseas Stanford, Princeton, Arizona State University, the Berkeley Collegeof Music, and the California Institute for the Arts. These courses canattract hundreds of thousands of students each. In some cases, coursesare offered free of charge. In some cases, new educational businessmodels are being developed, including models in which students may becharged for deeper evaluation and/or credit, or in which advertisingprovides a revenue stream.

While some universities have created their own Learning ManagementSystems (LMS), a number of new companies have begun organizing andoffering courses in partnership with universities or individuals.Examples of these include Coursera, Udacity, and edX. Still othercompanies, such as Moodle and Blackboard, offer LMS designs and servicesfor universities who wish to offer their own courses.

Students taking on-line courses usually watch video lectures, engage inblog/chat interactions, and submit assignments, exercises, and exams.Submissions may be evaluated (to lesser or greater degrees, depending onthe type of course and nature of the material), and feedback on qualityof coursework submissions can be provided. While many courses areoffered that evaluate submitted assignments and exercises, the natureand mechanics of the evaluations are generally of four basic types:

-   -   1) In some cases, human graders evaluate the exercises,        assignments, and exams. This approach is labor intensive, scales        poorly, can have consistency/fairness problems and, as a general        proposition, is only practical for smaller online courses, or        courses where the students are (or someone is) paying enough to        hire and train the necessary number of experts to do the        grading.    -   2) In some cases, assignments and exams are crafted in        multiple-choice, true false, or fill-in-the blank style, such        that grading by machine can be easily accomplished. In some        cases, the grading can be instant and interactive, helping        students learn as they are evaluated, and possibly shortening        the exam time, e.g., by guiding students to harder/easier        questions based on responses. However, many types of subject        matter, particularly those in which artistic expression or        authorship are involved, do not lend themselves to such        assignment or examination formats.    -   3) In some cases, researchers have developed techniques by which        essay-style assignments and/or exams may be scanned looking for        keywords, structure, etc. Unfortunately, solutions of this type        are, in general, highly dependent on the subject matter, the        manner in which the tests/assignment are crafted, and how        responses are bounded.    -   4) In some cases, peer-grading or assessment has been used,        whereby a student is obligated to grade the work of N other        students. Limitations of, and indeed complaints with,        peer-assessment include lack of reliability, expertise and/or        consistency. Additional issues include malicious or spiteful        grading, general laziness of some students, drop-outs and the        need to have students submit assignments at the same time,        rather than at individual paces.

Improved techniques are desired, particularly techniques that arescalable to efficiently and consistently serve large student communitiesand techniques that may be employed in subject matter areas, such asartistic expression, creative content computer programming and evensignal processing, that have not, to-date, proved to be particularlyamenable to conventional machine grading techniques.

SUMMARY

For courses that deal with media content, such as sound, music,photographic images, hand sketches, video (including videos of dance,acting, and other performances, computer animations, music videos, andartistic video productions), conventional techniques for automaticallyevaluating and grading assignments are generally ill-suited to directevaluation of coursework submitted in media-rich form. Likewise, forcourses whose subject includes programming, signal processing or otherfunctionally-expressed designs that operate on, or are used to producemedia content, conventional techniques are also ill-suited. Instead, ithas been discovered that media-rich, indeed even expressive, content canbe accommodated as, or as derivatives of, coursework submissions usingfeature extraction and machine learning techniques. In this way, e.g.,in on-line course offerings, even large numbers of students and studentsubmissions may be accommodated in a scalable and uniform grading orscoring scheme. Instructors or curriculum designers may adaptivelyrefine their assignments or testing based on classifier feedback. Usingthe developed techniques, it is possible to administer courses andautomatically grade submitted work that takes the form of mediaencodings of artistic expression, computer programming and even signalprocessing to be applied to media content.

In some embodiments in accordance with the present invention(s), amethod is provided for use in connection with automated evaluation ofcoursework submissions. The method includes receiving from an instructoror curriculum designer a selection of exemplary media content to be usedin evaluating the coursework submissions. The exemplary media contentincludes a training set of examples each assigned at least one qualityscore by the instructor or curriculum designer. The method furtherincludes accessing computer readable encodings of the exemplary mediacontent that together constitute the training set and extracting fromeach instance of exemplary media content a first set of computationallydefined features. The method includes, for each instance of exemplarymedia content, supplying a classifier with both the instructor orcurriculum designer's assigned quality score and values for thecomputationally defined features extracted therefrom and, based on thesupplied quality scores and extracted feature values, training theclassifier, wherein the training includes updating internal statesthereof. The method further includes accessing a computer readableencoding of media content that constitutes, or is derived from, thecoursework submission and extracting therefrom a second set ofcomputationally defined features, applying the trained classifier to theextracted second set of computationally defined features and, basedthereon, assigning a particular quality score to the courseworksubmission.

In some cases or embodiments, plural additional classifiers are suppliedwith respective instructor or curriculum designer's assigned qualityscores and values for computationally defined features extracted fromrespective instances of the exemplary media content. The additionalclassifiers are trained and applied as before. In some cases orembodiments, the quality score is, or is a component of, a grading scalefor an assignment- or test question-type coursework submission.

In some cases or embodiments, the media content that constitutes, or isderived from, the coursework submission includes an image or videosignal encoding and the method further includes (i) segmenting the imageor video frame to define at least one region of interest, and (ii)extracting, for the segmented at least one region of interest, aparticular computationally defined feature of the second set. In somecases or embodiments, the particular computationally defined featurecharacterizes either or both of (i) size or position of the region ofinterest relative to a rule of thirds alignment or point; and (ii) focusof, or within, the region of interest relative at least one otherportion of the image or video frame. In some cases or embodiments, thesegmenting provides foreground/background segmentation. In some cases orembodiments, the segmenting includes adaptively refined keypoint sets,wherein adaptive refinement is based on at least one of: iteration onthe keypoint sets using automatically updated keypoint detectionparameters; and computational clustering to refine the keypoint sets.

In some cases or embodiments, the coursework submission includessoftware code submitted in satisfaction of a programming assignment ortest question, the software code executable to perform, or compilable toexecute and perform, digital signal processing to produce output mediacontent. The exemplary media content includes exemplary output mediacontent produced using exemplary software codes, and the particularquality score assigned to the coursework submission is based on theapplying of the classifier to the second set of computationally definedfeatures extracted from the output media content produced by executionof the submitted software code.

In some cases or embodiments, the software code coursework submission isexecutable to perform digital signal processing on input media contentto produce the output media content, and the exemplary output mediacontent is produced from the input media content using the exemplarysoftware codes. In some cases or embodiments, the output media contentincludes audio signals processed or rendered by the software codecoursework submission.

In some cases or embodiments, the media content includes images or videoprocessed or rendered by the software code coursework submission. Insome cases or embodiments, the coursework submission includes a computerreadable media encoding of expressive media content selected from theset of: sketches, paintings, photographic images or other artistic stillvisuals; and synchronized audiovisual content, computer animation orother video that is itself expressive or visually captures underlyingexpression such as dance, acting, or other performance.

In some cases or embodiments, the media content that constitutes, or isderived from, the coursework submission includes an audio signalencoding, and for the first and second sets, at least some of thecomputationally defined features are selected or derived from: a rootmean square energy value; a number of zero crossings per frame; aspectral flux; a spectral centroid; a spectral roll-off measure; aspectral tilt; a mel-frequency cepstral coefficients (MFCC)representation of short-term power spectrum; a beat histogram; and/or amulti-pitch histogram computed over at least a portion of the audiosignal encoding.

In some cases or embodiments, the media content that constitutes, or isderived from, the coursework submission includes an image or videosignal encoding. For the first and second sets, at least some of thecomputationally defined features are selected or derived from: colorhistograms; two-dimensional transforms; edge, corner or ridgedetections; curve or curvature features; a visual centroid; and/oroptical flow computed over at least a portion of the image or videosignal encoding.

In some cases or embodiments, the classifier implements an artificialneural network (NN), k-nearest neighbor (KNN), Gaussian mixture model(GMM), support vector machine (SVN) or other statistical classificationtechnique.

In some embodiments, the method further includes iteratively refiningthe classifier training based on supply of successive instances of theexemplary media content to the classifier and updates to internal statesthereof. In some embodiments the method further includes continuing theiterative refining until an error metric based on a current state ofclassifier training falls below a predetermined or instructor orcurriculum designer-defined threshold.

In some cases or embodiments, the classifier is implemented using one ormore logical binary decision trees, blackboard voting-type methods, orrule-based classification techniques.

In some embodiments the method further includes supplying the instructoror curriculum designer with an error metric based on a current state ofclassifier training. In some embodiments the method further includessupplying the instructor or curriculum designer with a coursework taskrecommendation based on a particular one or more of the computationallydefined features that contribute most significantly to classifierperformance against the training set of exemplary media content.

In some cases or embodiments, the first and second sets ofcomputationally defined features are the same. In some cases orembodiments, the second set of computationally defined features includesa subset of the first set features selected based on contribution toclassifier performance against the training set of exemplary mediacontent. In some cases or embodiments, the quality score is, or is acomponent of a grading scale for an assignment- or test question-typecoursework submission. In some embodiments, the method further includesreceiving from the instructor or curriculum designer at least an initialdefinition of the first set of computationally defined features.

In some embodiments in accordance with the present invention(s), acomputational system including one or more operative computersprogrammed to perform the method of any of foregoing methods. In somecases or embodiments, the computational system is itself embodied, atleast in part, as a network deployed coursework submission system,whereby a large and scalable plurality (>50) of geographically dispersedstudents may individually submit their respective coursework submissionsin the form of computer readable information encodings. In some cases orembodiments, the computational system includes a student authenticationinterface for associating a particular coursework submission with aparticular one of the geographically dispersed students.

In some embodiments in accordance with the present invention(s),non-transient computer readable encoding of instructions executable onone or more operative computers to perform any of the foregoing methods.

In some embodiments in accordance with the present invention(s), acoursework management system for automated evaluation of courseworksubmissions includes an instructor or curriculum designer interface, atraining subsystem and a coursework evaluation deployment of a trainedclassifier. The instructor or curriculum designer interface selects orreceives exemplary media content to be used in evaluating the courseworksubmissions. The exemplary media content includes a training set ofexamples each assigned at least one quality score by the instructor orcurriculum designer. The training subsystem is coupled and programmed toaccess computer readable encodings of the exemplary media content thattogether constitute the training set and to extracting from eachinstance of exemplary media content a first set of computationallydefined features. The training subsystem is further programmed to, foreach instance of exemplary media content, supply a classifier with boththe instructor or curriculum designer's assigned quality score andvalues for the computationally defined features extracted therefrom, andto, based on the supplied quality scores and extracted feature values,train the classifier, wherein the training includes updating internalstates thereof. The coursework evaluation deployment of the trainedclassifier is coupled and programmed to access a computer readableencoding of media content that constitutes, or is derived from, thecoursework submissions and to extract therefrom a second set ofcomputationally defined features. The coursework evaluation deploymentapplies the trained classifier to the extracted second set ofcomputationally defined features and, based thereon, assigns aparticular quality score to the coursework submission.

In some cases or embodiments, the training subsystem supplies pluraladditional classifiers with respective instructor or curriculumdesigner's assigned quality scores and values for computationallydefined features extracted from respective instances of the exemplarymedia content and trains the additional classifiers. The courseworkevaluation deployment also applies the trained additional classifiers.

In some cases or embodiments, the coursework management system furtherincludes an execution environment. The coursework submission includessoftware code submitted in satisfaction of a programming assignment ortest question. The software code is executable in the executionenvironment to perform, or compilable to execute in the executionenvironment and perform, digital signal processing to produce outputmedia content. The exemplary media content includes the output mediacontent produced using the submitted software code. The particularquality score assigned to the coursework submission is based on theapplying of the classifier to the second set of computationally definedfeatures extracted from the output media content produced using thesubmitted software code. In some cases or embodiments, the output mediacontent includes audio signals processed or rendered by the softwarecode coursework submission.

In some cases or embodiments, the classifier implements an artificialneural network (NN), k-nearest neighbor (KNN), Gaussian mixture model(GMM), support vector machine (SVN) or other statistical classificationtechnique. In some cases or embodiments, the training subsystem allowsthe instructor or curriculum designer to iteratively refine theclassifier training based on supply of successive instances of theexemplary media content to the classifier and updates to internal statesthereof. In some cases or embodiments, the classifier is implementedusing one or more logical binary decision trees, blackboard voting-typemethods, or rule-based classification techniques.

In some embodiments in accordance with the present invention(s), acoursework management system includes means for selecting or receivingexemplary media content to be used in evaluating the courseworksubmissions, the exemplary media content including a training set ofexamples each assigned at least one quality score by the instructor orcurriculum designer; means for extracting from each instance ofexemplary media content a first set of computationally defined features,for supplying a classifier with both the instructor or curriculumdesigner's assigned quality score and values for the computationallydefined features extracted therefrom and, based on the supplied qualityscores and extracted feature values, for training the classifier; andmeans for extracting from the coursework submissions a second set ofcomputationally defined features, for applying the trained classifier tothe extracted second set of computationally defined features and, basedthereon, for assigning a particular quality score to the courseworksubmission.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings.

FIG. 1 depicts an illustrative networked information system in whichstudents and instructors (and/or curriculum developers) interact withcoursework management systems.

FIG. 2 depicts data flows, interactions with, and operationaldependencies of, various components of a coursework management systemthat provides automated coursework evaluation in accordance with someembodiments of the present invention(s).

FIG. 3 depicts both instructor-side and student-side portions of afeature extraction and machine learning system process flow formedia-rich assignments or examinations in accordance with someembodiments of the present invention(s).

FIG. 4 is a flow diagram of a process in accordance with someembodiments of the present invention(s) by which certaincomputationally-defined features are extracted from an image or frame inmotion video.

FIG. 5 is a flow diagram that illustrates successive computational stepsperformed in accordance with some embodiments of the presentinvention(s) to automatically segment foreground and background of animage.

FIG. 6 depicts positions and regions in an image field consistent with a“rule of thirds” principle and as an illustration for understanding ofrelated computational techniques described herein.

FIG. 7 is a flow diagram that illustrates automatic assessment, inaccordance with some embodiments of the present invention(s), of whethera photo (or other visual work) adheres to a “rule of thirds” principle.

FIGS. 8A, 8B, 8C and 8D illustrate, by way of a set of images, a processin accordance with some embodiments of the present invention(s) by whichan image (e.g., FIG. 8A) is segmented into foreground and backgroundcomponents (see FIGS. 8B and 8C) and by which a foreground region ofinterest is tested (see FIG. 8D) against a “rule of thirds” principle.

FIGS. 9A, 9B, 9C and 9D illustrate, using exemplary images, focus and/ordepth of field characteristics pertinent to certain focus detection andsegmentation techniques described herein and employed in accordance withsome embodiments of the present invention(s).

FIG. 10 is a flow diagram that illustrates computational techniques, inaccordance with some embodiments of the present invention(s), fordetermining depth of field by comparing foreground and background focusmeasures or by comparison with a reference image.

FIG. 11 is a flow diagram that illustrates computational techniques, inaccordance with some embodiments of the present invention(s), forexamining depth of field spatially to determine directionality or bygrading against a reference image.

The use of the same reference symbols in different drawings indicatessimilar or identical items.

DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

The computational techniques described herein address practicalchallenges associated with administration of educational courses ortesting, including on-line courses offered for credit to large andgeographically dispersed collections of students (e.g., over theInternet), using advanced feature extraction techniques combined withmachine learning (ML) algorithms. The developed techniques areparticularly well-suited to educational or testing domains in whichassignments or test problems call for expressive content, such as sound,music, photographic images, hand sketches, video (including videos ofdance, acting, and other performances, computer animations, musicvideos, and artistic video productions). The developed techniques arealso well-suited to educational or testing domains in which assignmentsor test problems include programming, signal processing or otherfunctionally expressed designs that operate on, or are used to producemedia content, and which may be evaluated based on qualities of themedia content itself. In each of the foregoing cases, conventionaltechniques for automatically evaluating and grading assignments(typically multiple choice, true/false or simple fill-in-the-blank orshort answer questions) are ill-suited to the direct evaluation ofcoursework that takes on media-rich forms.

Instead, it has been discovered that media-rich, indeed even expressive,content can be accommodated as, or as derivatives of, courseworksubmissions using feature extraction and machine learning techniques. Inthis way, e.g., in on-line course offerings, even large numbers ofstudents and student submissions may be accommodated in a scalable anduniform grading or scoring scheme. Instructors or curriculum designersare provided with facilities to adaptively refine their assignments ortesting based on classifier feedback. Using the developed techniques, itis possible to administer courses and automatically grade submitted workthat takes the form of media encodings of artistic expression, computerprogramming and even signal processing to be applied to media content.

Illustrative System(s) for Automated Coursework Evaluation

FIG. 1 depicts an illustrative networked information system in whichstudents and instructors (and/or curriculum developers) interact withcoursework management systems 120. In general, coursework managementsystems 120 such as described herein may be deployed (in whole or inpart) as part of the information and media technology infrastructure(networks 104, servers 105, workstations 102, database systems 106,including e.g., audiovisual content creation, design and manipulationsystems, code development environments, etc. hosted thereon) of aneducational institution, testing service or provider, accreditationagency, etc. Coursework management systems 120 such as described hereinmay also be deployed (in whole or in part) in cloud-based orsoftware-as-a-service (SaaS) form. Students interact with audiovisualcontent creation, design and manipulation systems, code developmentenvironments, etc. deployed (in whole or in part) on user workstations101 and/or within the information and media technology infrastructure.In many cases, audiovisual performance and/or capture devices (e.g.,still or motion picture cameras 191, microphones 192, 2D or 3D scanners,musical instruments, digitizers, etc.) may be coupled to or accessed by(or from) user workstations 101 in accordance with the subject matter ofparticular coursework and curricula.

In some cases, the audiovisual content created using such performanceand/or capture devices constitutes, or forms a substantial component of,a student's coursework submission. For example, the courseworksubmission may include an encoding of an audio signal captured inconnection with a vocal performance or a still image or motion videocaptured using a camera. Likewise, the coursework submission may includean encoding of an audio signal or image or video content synthesized orcreated in the first instance by the student using computer-based, mediacontent creation, design and/or manipulation systems. In some cases, astudent's coursework submission may constitute, or include, audiovisualcontent derived from some primary media content (e.g., one or more audiosignals, image or video content, etc.) supplied as part of an assignmentor test question. For example, a coursework submission might include anencoding of an audio signal that demonstrates the student's skill inmixing sources, equalizing, panning and/or compressing signals, etc.based on some basic audio encodings supplied as part of the assignmentor test question. Likewise, a coursework submission might includeencoding of an image or motion video that demonstrates the student'sskill in placing, arranging or sizing subject matter elements inaccordance with compositional figures of merit, performing colorcorrections, using or harmonizing colors, establishing mood by use ormanipulation of hue, contrast or saturation, creating visual flow ormanaging scene changes.

FIG. 2 depicts data flows, interactions with and operationaldependencies of various components of an instance of courseworkmanagement system 120 that includes an automated coursework evaluationsubsystem 121 in accordance with some embodiments of the presentinvention(s). In particular, automated coursework evaluation subsystem121 includes a training/courseware design component 122 and a courseworkevaluation component 123. An instructor and/or curriculum designer 202interacts with the training/courseware design component 122 to establish(for given coursework such as a test, quiz, homework assignment, etc.) agrading rubric (124) and to select related computationally-definedfeatures (124) that are to be used to characterize quality or scoring(e.g., in accordance with criteria and/or performance standardsestablished in the rubric or ad hoc) for coursework submissions bystudents.

For example, in the context of an illustrative audio processingassignment, a rubric may define criteria including distribution of audioenergy amongst selected audio sub-bands, degree or quality ofequalization amongst sub-bands, degree of panning for mixed audiosources and/or degree or quality of signal compression achieved by audioprocessing. In the context of an illustrative image or video processingassignment, a rubric may define criteria for tonal or chromaticdistributions, use of focus or depth of field, point of interestplacement, visual flow and/or quality of image/video compressionachieved by processing. Based on such rubrics, or in accord with ad hocselections by instructor and/or curriculum designer 202, particularcomputationally-defined features are identified that will be extracted(typically) based on signal processing operations performed on mediacontent (e.g., audio signals, images, video, digitized 3D surfacecontours or models, etc.) and used as input feature vectors in acomputational system implementation of a classifier. Instructor and/orcurriculum designer 202, also supplies (or selects) media contentexemplars 126 and scoring/grading 127 thereof to be used in classifiertraining 125.

In general, any of a variety of classifiers may be employed inaccordance with statistical classification and other machine learningtechniques that exhibit acceptable performance in clustering orclassifying given data sets. Suitable and exemplary classifiers areidentified herein, but as a general proposition, in the art of machinelearning and statistical methods, an algorithm that implementsclassification, especially in concrete and operative implementation, iscommonly known as a “classifier.” The term “classifier” is sometimesalso used to colloquially refer to the mathematical function,implemented by a classification algorithm that maps input data to acategory. For avoidance of doubt, a “classifier,” as used herein, is aconcrete implementation of statistical or other machine learningtechniques, e.g., as one or more of code executable on one or moreprocessors, circuitry, artificial neural systems, etc. (individually orin combination) that processes instances explanatory variable data(typically represented as feature vectors extracted from instances ofdata) and groups the instances into categories based on training sets ofdata for which category membership is known or assigned a priori.

In the terminology of machine learning, classification can be consideredan instance of supervised learning, i.e., learning where a training setof correctly identified observations is available. A correspondingunsupervised procedure is known as clustering or cluster analysis, andtypically involves grouping data into categories based on some measureof inherent statistical similarity uninformed by training (e.g., thedistance between instances, considered as vectors in a multi-dimensionalvector space). In the context of the presently claimed invention(s),classification is employed. Classifier training is based on instructorand/or curriculum designer inputs (exemplary media content andassociated grading or scoring), feature vectors used characterize datasets are selected by the instructor or curriculum designer (and/or insome cases established as selectable within a training/courseware designmodule of an automated coursework evaluation system), and data sets are,or are derived from, coursework submissions of students.

Based on rubric design and/or feature selection 124 and classifiertraining 125 performed (in training/courseware design component 122)using instructor or curriculum designer 202 input, feature extractiontechniques and trained classifiers 128 are deployed to courseworkevaluation component 123. In some cases, a trained classifier isdeployed for each element of an instructor or curriculum designerdefined rubric. For example, in the audio processing example describedabove, trained classifiers may be deployed to map each of the following:(i) distribution of audio energy amongst selected audio sub-bands, (ii)degree or quality of equalization amongst sub-bands, (iii) degree ofpanning for mixed audio sources and (iv) degree or quality of signalcompression achieved by audio processing to quality levels or scoresbased on training against audio signal exemplars. Likewise, in theimage/video processing example described above, trained classifiers maybe deployed to map each of the following: (i) distribution of tonal orchromatic values, (ii) focus or depth of field metrics, (iii)positioning or flow with a visual field of computationally discerniblepoints/regions of interest and (iv) degree or quality of image/videocompression to quality levels or scores based on training against imageor video content exemplars. In some cases, features extracted frommedia-rich content 111 that constitutes, or is derived from, courseworksubmissions 110 by students 201 are used as inputs to multiple of thetrained classifiers. In some cases, a single trained classifier may beemployed, but more generally, outputs of multiple trained classifiersare mapped to a grade or score (129), often in accordance with curvespecified by the instructor or curriculum designer.

Resulting grades or scores 130 are recorded for respective courseworksubmissions and supplied to students 201. Typically, courseworkmanagement system 120 includes some facility for authenticatingstudents, and establishing, to some reasonable degree of certainty, thata particular coursework submission 110 is, in fact, submitted by thestudent who purports to submit it. Student authentication may beparticularly important for course offered for credit or as a conditionof licensure. While student authentication is not essential to allcoursework management system implementations that provide automatedcoursework evaluation in accord with embodiments of the presentinvention(s), suitable student authentication techniques are detailed incommonly-owned, Provisional Application No. 62/000,522, filed May 19,2014, entitled “MULTI-MODAL AUTHENTICATION METHODS AND SYSTEMS” andnaming Cook, Kapur, Vallis and Hochenbaum as inventors, the entirety ofwhich is incorporated herein by reference.

In some embodiments of coursework management system 120 (see e.g., FIG.2), an automated coursework evaluation subsystem 121 may cooperate withstudent authentication facilities, such as fraud/plagiarism detection.For example, if coursework submissions (ostensibly from different,separately authenticated students) exhibit exactly or nearly the samescore(s) based on extracted computationally defined features andclassifications, then fraud or plagiarism is likely and can be noted orflagged for follow-up investigation. Likewise, if a courseworksubmission exhibits exactly the same score(s) (again based on extractedcomputationally defined features and classifications) as a gradingexemplar or model audio signal, image, video or other expressive mediacontent supplied to the students as an example, then it is likely thatthe coursework submission is, in-fact, a submission of the example,rather than the student's own work. Based on the description herein,persons of skill in the art will appreciate these and other benefits ofintegrating student authentication and automated coursework evaluationfacilities in some embodiments of a coursework management system.

Exemplary Use Cases

The developed techniques provide instructors and curriculum designerswith systems and facilities to add training examples together withgrading or scoring characterizations thereof. For example, in some casesor embodiments, an instructor or curriculum designer 202 may identifycertain training examples as exemplars (e.g., exemplars 126, see FIGS.2, 3) of good and bad (or good, mediocre, and bad) courseworksubmissions that have been scored/graded (or that the instructor orcurriculum designer may score/grade) 127. The system extracts (125A)computationally-defined features from the training examples, and usesthese extracted features to train (125) a computational system thatimplements a classifier.

In general, an operative set of feature extractors may be interactivelyselected or defined (see rubric design/feature selection 124, FIG. 2 andselect features/define decision logic 324, FIG. 3) for a particularassignment or test question. In some cases, systems or methods inaccordance with the present inventions provide (and/or guide) theinstructor or curriculum designer through a menu or hierarchy of featureextractor selections and/or classification stages. In some cases, adecision tree of rules may be automatically derivable from the providedfiles of good/bad examples. In some cases, it may be desirable to allowthe instructor/curriculum designer to identify (or label) what is goodor bad about the training examples. For example, in an application ofthe developed techniques to grading of music coursework submissions,systems and methods may allow the instructor/curriculum designer to notethat a training example is (1) in the key of C, (2) has more than 2, butless than 10, discernible sections, (3) has a very strong beat, etc. Inan application of the developed techniques to grading of imagecoursework submissions, systems and methods may allow theinstructor/curriculum designer to note that a training example (1)places an in-focus foreground element in accordance with the rule ofthirds, (2) employs a shallow depth of field, discernible sections, (3)uses hue, contrast and saturation in a manner that presents in a “filmnoir” style or mood.

The classifier learns to categorize submissions in accordance with theinstructor or curriculum designer's classifications (e.g., on a gradingscale or against a rubric), and (at least during training) provides theinstructor or curriculum designer with feedback (203) as to how well(statistically speaking) submissions will be classified based on thecurrent training. In some cases, the system makes suggestions as to howto change the task or criteria so that submissions are easier toclassify and thus grade. In response, the instructor or curriculumdesigner can modify the assignment or evaluation criteria, resubmittingthe original examples or modified ones, until they (and the system) aresatisfied that the system will perform well enough for grading studentsubmissions. As illustrated in FIG. 3, at least some embodiments ofcoursework management system 120, provide an iterative and interactiveinstructor (or curriculum designer) interaction to identify the set ofcomputationally defined features and to train classifiers for deploymentas an automated evaluator (see feature extraction and trainedclassifiers 128) of coursework submissions suitable for studentinteractions or use by a course administrator.

“Good” exemplars 126 (i) can come from historical or current masters inthe field or can be examples that are representative of the style beingemulated, (ii) can be generated by the curriculum designer or (iii) caninclude previous student submissions from prior administrations of thecourse (or even hand-picked grading exemplars from a currentadministration of the course). In some cases or educational domains,initial “bad” exemplars 126 can be provided by the curriculum designeror drawn from student submissions (whether from prior administrations orcurrent exemplars). In some cases, once the system is used to offer acourse or evaluate an assignment once, prior training (125) serves as abaseline and hand-selected student submissions are thereafter used tore-train the system, or refine the training, for better results.

Aside from speed and convenience, and the ability to evaluate thousandsrather than tens of submissions in a short time can provide significantadvantages. Furthermore, in some cases or embodiments, computationalcapabilities of the classifier may be scaled as needed, e.g., bypurchasing additional compute power via cloud services or compute farms.Additional benefits include absolute objectivity and fairness. Allassignments can be evaluated by exactly the same rules and trainedmachine settings. This is not the case with a collection of humanexaminers, who inevitably bring biases and grow fatigued during/betweensessions, yielding inconsistent results.

Often, coursework submissions 111 are presented for automated evaluationas computer readable media encodings uploaded or directly selected bystudents from their respective workspaces. In some cases, a courseadministrator may act as an intermediary and present the courseworksubmissions 111 to the automated coursework evaluation subsystem 121.Suitable encoding formats are dependent on the particular media contentdomain to which techniques of the present invention are applied and are,in general, matters of design choice. Nonetheless, persons of skill inthe art, having benefit of the present disclosure will appreciate use ofsuitable data access methods and/or codecs to obtain and present mediacontent in data structural forms that are, in turn, suitable orconvenient for feature extraction and use as classifier inputs in anyparticular implementation of the techniques described herein.

Media-Rich Grading Examples:

Persons of skill in the art having access to the present disclosure willappreciate a wide range of media-rich content for which the describedautomated grading techniques may be applied or adapted. Nonetheless, asa set of illustrative examples, we summarize computationally-definedfeatures and mappings performed by a trained classifier first forseveral aspects of an audio processing course rubric and then for amusic programming course rubric. Sets of computationally-defined audiofeatures suitable to such courses are also summarized. Finally, weoutline an application of similar techniques to media contentcharacteristic of visual art, still images and/or motion video.

Audio Processing Course Example:

One illustrative example of a media-rich educational domain in whichtechniques of the present invention may be employed is audio processing,e.g., application of digital signal processing techniques to audiosignal encodings. Depending on the course, operative implementations ofsuch techniques may be made available to students in the form of audioprocessing systems, devices and/or software, or as an audio processingtoolset, library, etc. Students may learn to use these operativeimplementations of signal processing techniques to manipulate andtransform audio signals.

For example, in a basic audio processing course, students may be taughtaudio composition, sub-band equalization, mixing and panning techniques,use and introduction of reverberation, signal compression, etc. In sucha course, students may be given assignments or quizzed or tested formastery of these audio processing techniques. A coursework managementsystem that provides automated evaluation of coursework submissions asdescribed herein may facilitate administration of such a course.Accordingly, based on the description herein, persons of skill in theart will appreciate that systems and techniques detailed above withreference to FIGS. 2 and 3 may be employed or adapted to evaluatecoursework submissions that seek to demonstrate student mastery oftechniques for manipulating and transforming a reference audio signal.For such a course, a rubric may specify grading of one or moretransformed audio signals for levels, equalization, panning andcompression.

Grading for Levels and Equalization:

To facilitate grading for levels (and equalization), an instructor orcurriculum designer may select or specify computational-defined featuresthat include calculations of RMS power in a transformed audio signalsubmitted by the student and in various mix-motivated sub-bands thereof.For example, in some cases or situations, RMS power may be calculated ineach of the following mix-motivated sub-bands for the submitted audiosignal:

<40 Hz (sub)

40-120 Hz (bass)

120-400 Hz (low-mid)

400-900 Hz (mid)

0.9-2.5 kHz (high-mid)

2.5-6 kHz (presence)

6-10 kHz (bite)

>10 kHz (air/sibilance)

Using such computational-defined features and/or derived mean/variancemeasures, a classifier (or classifiers) may be trained usingscored/graded reference signal exemplars to identify course worksubmissions that exhibit good, mediocre, and bad leveling from acompositional perspective. Likewise, a classifier (or classifiers) maybe trained using scored/graded reference signal exemplars to identifycourse work submissions that exhibit good, mediocre, and badequalization of sub-band levels. As will be appreciated, scoringquantization as good, mediocre, and bad is merely illustrative.

Grading for Panning:

To facilitate grading for panning, an instructor or curriculum designermay select or specify features that are computational-defined asfollows:

-   -   1) Time-align student submitted audio signal with a reference        audio signal and slice into beat-aligned segments.    -   2) Sum the energy in each segment in 24 bark-frequency bands,        producing a beat-bark matrix.    -   3) Calculate channel similarity between Left and Right channels        of submitted student audio signal as

${\psi\left( {m,k} \right)} = \frac{\left| {{X_{L}\left( {m,k} \right)} \cdot {X_{R}^{*}\left( {m,k} \right)}} \right|}{\left| {X_{L}\left( {m,k} \right)} \middle| {}_{2}{+ \left| {X_{R}\left( {m,k} \right)} \right|^{2}} \right.}$

-   -   -   (where m is bark index and k is beat index).

    -   4) Calculate partial similarity measures for each channel as        e.g.

${{\psi_{L}\left( {m,k} \right)} = \frac{\left| {{X_{L}\left( {m,k} \right)} \cdot {X_{R}^{*}\left( {m,k} \right)}} \right|}{\left| {X_{L}\left( {m,k} \right)} \right|^{2}}},{{\psi_{R}\left( {m,k} \right)} = {{\frac{\left| {{X_{R}\left( {m,k} \right)} \cdot {X_{L}^{*}\left( {m,k} \right)}} \right|}{\left| {X_{R}\left( {m,k} \right)} \right|^{2}}{\Psi_{L}\left( {m,k} \right)}} = \frac{\left| {{X_{L}\left( {m,} \right)} \cdot {X_{R}^{*}\left( {m,k} \right)}} \right|}{\left| {X_{L}\left( {m,k} \right)} \right|^{2}}}}$

-   -   -   These partial similarities are then used to produce a sign            function, i.e., −1 if ψ_(L) (m, k)>ψ_(R) (m, k) and +1 if            vice versa.

    -   5) Calculate a final panning index as 1−ψ(m, k) times the sign        function.

    -   6) Apply a logistic mask based on signal energy to panning index        values to remove contribution from bins with little or no signal        energy.

    -   7) Calculate an overall panning score as the total absolute        difference between student panning index and reference panning        index over all beat-bark bins.        Using such computational-defined features, a classifier (or        classifiers) may be trained using scored/graded reference signal        exemplars to identify course work submissions that exhibit good,        mediocre, and bad panning. As before, scoring quantization as        good, mediocre, and bad is merely illustrative.

Grading for Compression:

To facilitate grading for compression, an instructor or curriculumdesigner may select or specify features that are computational-definedas follows:

-   -   1) Time-align student submitted audio signal with a reference        audio signal.    -   2) Calculate time-domain amplitude envelope of submitted student        file by low-pass filtering the squared amplitude (e.g., using a        512-order finite impulse response filter, 40 Hz cutoff, applied        forward and then reverse), then taking the square root of        half-wave rectified filtered signal.    -   3) Calculate a difference function between the envelope of the        student submitted audio signal and the reference.    -   4) Compute the average zero-crossing rate, average absolute        value, and average absolute first-order difference of the        difference function.        Using such computational-defined features, a classifier (or        classifiers) may be trained using scored/graded reference signal        exemplars to identify course work submissions that exhibit good,        mediocre, and bad compression. As before, scoring quantization        as good, mediocre, and bad is merely illustrative.

Grading for Compositional Effort:

Still another example is grading for “compositional effort” (in essence,a computational-defined feature, decision-tree, and classifier-basedevaluation of the question “is this music interesting?”). To evaluatecompositional effort, we extract features, use them (plus clustering ofwindows) to segment the submitted audio into sections. Decision treelogic can determine whether the number, and indeed structure, ofsections meets objectives specified in the grading rubric. In somecases, we can also compare sections pairwise usingcomputationally-defined features to determine structure (e.g., verse,chorus, verse, chorus, bridge, chorus), and scoring or grading can beassigned based on such determined structure.

Video Processing Course Examples:

Another illustrative example of a media-rich educational domain in whichtechniques of the present invention may be employed is image or videoprocessing, e.g., application of digital signal processing techniques tostill image or motion video content. Depending on the course, operativeimplementations of such techniques may be made available to students inthe form of image or video processing systems, devices and/or software,or as an image/video processing toolset, library, etc. Students maylearn to use these operative implementations of signal processingtechniques to manipulate and transform still image or motion videocontent.

Grading for Composition.

Composition generally describes an evaluation of the aesthetics of anarrangement of subject matter elements within a picture or image field.Automated techniques for grading for composition may therefore seek tocomputationally define and automate (in practical application)evaluations that embrace many of the core principles in photography,video, and other visual arts. In an effort to computationally assesscertain illustrative aspects of composition in visual works, we havedeveloped concrete computational mechanisms to automatically determine:

1. Foreground and Background

2. Center/point of interest within an image

3. Subject placement

4. Depth of field

FIG. 4 illustrates a top-level flow of a process by which certaincomputationally-defined features are extracted from an image 401 (oranalogously, from a frame in motion video). Based on operation ofvarious mechanisms 402 for detecting points or regions of interest,certain location/shape metrics or features 411 and depth of fieldfeatures 412 are computed (e.g., as part of the operation of featureextraction and trained classifiers 128, recall FIG. 2). Certainillustrative mechanisms 402 for detecting points or regions of interestare now detailed; however, based on the description herein, persons ofskill in the art having benefit of the present disclosure willappreciate a wide variety of additional mechanism (and/or adaptations ofmechanisms detailed herein) that may be utilized or incorporated in thecontext of automated coursework evaluation systems described herein.

Determining what is in the foreground and what is the background is animportant initial element of other techniques such as the determinationsof center/point/region of interest, depth of field, etc. describedherein. FIG. 5 illustrates successive computational steps performed infurtherance of techniques that have been developed to automaticallysegment the foreground and background of an image. In particular, webegin with a preprocessing pipeline 511 which converts image 401 tograyscale (521), reducing its dimensions (522), and finally, an optionalstep of illumination processing (523). Good results, and specificallyincreased segmentation efficacy, have been achieved using Tan-Triggsillumination normalizing techniques as part of illumination processing523.

After preprocessing (511), we perform keypoint detection (512) toidentify a general area of interest in which we believe the foregroundexists. Image keypoint detection is an active area of research in visionprocessing where images are decomposed into a sparse set of “features”,called keypoints (KP), representing points of interest. An idealkeypoint detector finds salient image regions (strong groups ofkeypoints) and is invariant to image transformations such astranslation, rotation, scaling, and affine deformation. In actualpractice, a keypoint detector seeks to approach the ideal and exhibit atleast substantial invariance to transformations typical in a particularimage or video processing domain.

In some embodiments of automated coursework evaluation systems describedherein, we run a keypoint detector adaptively to refine the set ofkeypoints returned. Currently, we use two different approaches foradaptive keypoint detection 524. The first approach is iterative,automatically updating its parameters so that we filter out extraneouskeypoints. The second approach uses a clustering method (e.g., K-Means),to refine the keypoints returned and come up with a subset of keypointscontaining the main foreground subject (largest cluster), or multiplesubjects of interests (n clusters which contain x % of the totalkeypoints). Keypoints are stored (525) for use in the segmentationpipeline 513.

After detecting (524), extracting and storing (525) a set of keypoints,we computationally determine (526) a primary region of interest (ROI)containing the main subject/foreground of the image. A coding oridentification of the primary region of interest is passed to asegmentation algorithm 527, which attempts to model the foreground inthe KP-derived ROI provided, as well as the background (everythingoutside the KP-derived ROI). One example of a segmentation algorithmthat we have found to be suitable in this context is the GrabCutalgorithm described in Rother, Carsten, Vladimir Kolmogorov, and AndrewBlake. “Grabcut: Interactive foreground extraction using iterated graphcuts.” ACM Transactions on Graphics (TOG) 8 Aug. 2004: 309-314. Ingeneral, a segmentation technique such as that implemented insegmentation algorithm 527 can return a number of outputs, includingmasks which are used to derive the final area of the foreground, and thefinal background areas. When keypoint detection 524 is run using theclustering approach, and multiple keypoint cluster regions are detected,operations in the segmentation pipeline 513 run iteratively, allowingfor multiple foreground subjects to be extracted from the rest of theimage.

Building then on a foreground/background segmentation such as in accordwith FIG. 5, many of the computationally-defined features or metricsintroduced in FIG. 4 may now be extracted. Turning first to acenter/point of interest and the “principle of thirds,” when oneprincipal subject or point of interest exists, it is generallyappreciated that, as a matter of composition or aesthetics, centerplacement of principal subject or point of interest in the center of thevisual field is not preferred, as it splits the picture in half, thusmaking it hard to balance. Accordingly, as illustrated in FIG. 6, theintersections 611 of lines that divide the picture area of image 401into thirds are typically considered (as a matter of compositionalanalysis) to be better positions for the point of interest in mostphotographs and other visual compositions.

As depicted in FIG. 7, it is possible to automatically assess if a photo(or other visual work) adheres to the “principle of thirds.” First, theforeground subject is extracted using the segmentation algorithm 520described above with reference to FIG. 5. Next, the principle of thirdsintersect points 611 (and corresponding image bisecting alignments) arecalculated (731) by dividing the image into thirds along the x and yaxes, and calculating positions. Then, a score is assigned (733) bydetermining (732) if the bounding area of the foreground subject(s)is/are non-center aligned and if the subject(s) intersect with theprinciple-of-thirds intersect points. A buffer area of some radius, ordefining a cost function, around the principle-of-thirds intersect pointcan be used. The sequence of images in FIGS. 8A-8D illustrate thisprocess, including segmenting original image (FIG. 8A) into foreground(FIG. 8B) and background (FIG. 8C) and testing against the “rule ofthirds” principal (FIG. 8D).

Two additional location or shape metrics or features 411 that are gradedautomatically are the position and size of the foreground subject. Thisis achieved by first segmenting foreground subject or point/region ofinterest (e.g., using segmentation techniques 520, 402 describedherein), and then determining the size/area and position of theforeground subject or point/region of interest within the image. Thesefeatures or metrics can be compared to a reference image, and graded tosee how closely the submitted image matches the composition of thereference image in terms of position and size. See generally, FIGS. 2, 3and the accompanying description above.

In the context of film and photography, depth of field (DOF) is sdistance between the nearest and farthest objects in a scene that appearacceptably sharp in an image. In some cases or for some compositions(and assignments), it may be desirable to have the entire image sharp,and a large DOF (or “deep focus”) is appropriate. For other compositionsor assignments, a small DOF (or “shallow focus”), may be more effective,emphasizing the subject while de-emphasizing the background, andsometimes parts of the foreground around the main subject.

Understanding and using DOF is at the core of photographic, film, andother visuals works and is therefore an important aspect that would bedesirable to address in a system that seeks to automatically evaluatecoursework submissions. Take, for example, the child in the image ofFIG. 9A. The subject (the child) is in focus while the background iscompletely out of focus—this is achieved using a shallow DOF, and theresult makes the child the point of interest in the image. While a humanmay visually perceive and cognitively react to the use of focus,techniques described herein seek to enable a computational system toevaluate this aspect of composition. In contrast with the first image,another image (FIG. 9B) of the apples has a slightly less shallow DOFresults in gradual decrease in focus over distance for an interestingvisual effect. The woman in the third image (FIG. 9C) is out of focuscompletely (blurry), while the screenshot (FIG. 9D) from Citizen Kaneuses deep focus to have the entire frame including all planes (mother atthe front, father at the door, young Kane in the window) in focus. Fromthese four examples, one can see how focus and DOF are usedcompositionally for many different effects, or even as a sign of poorimaging (as in the completely blurred image in FIG. 9C).

A variety of different measures for deducing focus in an image or partof an image will be appreciated by persons of skill in the art. One suchmeasure that is employed in some embodiments of automated courseworkevaluation systems described herein is a Cumulative Probability of BlurDetection (CPBD) algorithm such as described in Narvekar, Niranjan D,and Lina J. Karam. “A no-reference image blur metric based on thecumulative probability of blur detection (CPBD).” Image Processing, IEEETransactions on 20.9 (2011): 2678-2683.

One computational system approach determining and operating on depth offield, and for actually evaluating and grading image-based works, is tocalculate (1011) a focus metric for both foreground and backgroundsegments and then compare (1012) the focus measure of the foregroundsubject to that for the background of the frame (see FIG. 10, method 1).By calculating the difference between the foreground subject's focus andthe background, it is possible to determine if a shallow or deep DOF wasused. This is especially useful in many photographic and video-basedworks, where shallow depth of field might be used as a compositionaltechnique to make a subject pop out (as is often the case in portraitand still life photography), as exemplified by the images of FIG. 8A andFIG. 8B, or in the frame (FIG. 8D) from Citizen Kane where deep focus isused to provide multiple layers of complexity in the same image wherethe adults in the foreground are juxtaposed to young Kane in thebackground.

When depth of field (or more specifically, when a depth of fieldcompositional aspect to be evaluated by an automated courseworkevaluation systems) is in-between being fully shallow or fully deep(e.g., when lens aperture is between a largest and smallest setting), itmay not be useful to compare the focus of the foreground to thebackground. For example, the focus for the apples in FIG. 8B (which isexhibits relatively shallow depth of field) gradually decreases overdistance in the image. In such cases, it is possible to computationallycharacterize depth of field by splitting the image into regions ofinterest and calculating the focus measure over each region (see 1111,FIG. 11). This can be done along a particular axis, or along multipleaxes. For example, by computationally characterizing the focus as onemoves along the x- and y-axes, it is possible to determine that theimage gradually increases in focus as we reach the final apple, upfront, and at the right of the image, by comparing the focus measurefrom the left side toward the right. In fact, the apple is closest tothe lens, and the apples to the left and behind gradually decrease infocus as they get further away from the camera. Using this information,it is possible to calculate directionality and proximity (1112) and toultimately use these extracted features in classification. Likewise, thefocus over each ROI can be compared to a reference image, and a gradeassigned by taking the sum of the absolute differences (1122) betweenreference and submitted image (see FIG. 11, method 2). When a referenceimage is not used, the degree of DOF can be deduced by deducing thedirectionality and proximity from the focus map derived from calculatingthe focus over each ROI (see FIG. 11, method 1).

Next we provide examples of possible assignments which are “gradable”using the metrics and techniques described herein. Note that thissection is not intended to be exhaustive of all possibilities orcombinations of features or metrics, rather it provides an introductionto the ways in which the techniques presented can be used toautomatically assess image, video, and other visual based works in thecontext of education environments and automated coursework evaluationsystems.

Image Assignment: General Composition.

For this assignment a student is instructed to adhere to the basicprinciples of composition taught in the course (focus/sharpness,principle-of-thirds, depth-of-field). The submitted works are assessedusing techniques described herein (and a score assigned) for anycombination of the following criteria:

1. Overall focus (is the image in focus overall or is it blurry/out offocus)

2. Is there a defined point-of-interest(s) in the image

3. Does the image adhere to the principle-of-thirds

Alternatively, the assignment can include a reference image in which thestudent needs to analyze and recreate, compositionally. Note, this doesnot mean the subject and scene need to be the same—rather the size andplacement (including principle-of-thirds) of the foreground subject(s),and depth of field can be analyzed in the reference image, and comparedto the same metrics determined in the submitted work. A score can beassigned by how closely the student matched the reference work.

Image Assignment: Portrait Photography.

As exemplified with the boy in the image FIG. 8A, portrait photographyoften uses a shallow focus to highlight the person in the image whileblurring the background. In this more domain specific example, thistechnique can be automatically assessed using the segmentation algorithmdetailed earlier, and by either examining the difference between theforeground subject's (individual) and background's (scene) focusmeasures (larger difference would yield a higher score), or by lookingat the difference between the submitted work's foreground and areference work's foreground, and the submitted work's background focusand a reference work's background focus (and by assigning a score basedon how closely they match).

Video Assignment: Depth-of-Field & Manual Focus Effect/Transition.

In the evaluation of this assignment, previously described techniquesfor segmentation and extraction of depth-of-field features or metricsare used in combination with either scene-change detection orframe-by-frame analysis to analyze video/film works. For example, thestudent can be instructed to utilize a shallow focused shot (like theshot of the boy in FIG. 8A or the apples in FIG. 8B) in one scene with adeep focus shot in another (exemplified by the frame from Citizen Kanein FIG. 8D). In combination with techniques determining the location ofscene changes, the depth of field between scenes can be determined and ascore given. Additionally, utilizing frame-by-frame analysis enables notonly the detection/presence of these techniques or transitions, but alsotheir location and durations—any of which can be parameterized in anassignment and combined to assign an overall score or grade.

Grading for Color and Contrast.

Color and Contrast are used heavily in photography, video, and thevisual arts, for artistic and stylistic reasons, and also correctionalreasons (e.g. color correcting images/film for consistency, enhancingdepth or vibrance). In addition to composition aspects of visual works,we are able to analyze images and other visual works in a number ofways. Some of the features we are able to computationally define and usein automated coursework evaluation include:

-   -   1. Contrast Measure    -   2. Saturation Levels    -   3. Average Brightness    -   4. Average Hue    -   5. Color/Grayscale Image Histogram—“An image histogram is a type        of histogram that acts as a graphical representation of the        tonal distribution in a digital image. It plots the number of        pixels for each tonal value. By looking at the histogram for a        specific image a viewer will be able to judge the entire tonal        distribution at a glance.”    -   6. Color Quantization—Color Quantization can be a useful tool to        help simplify color content of an image for further color        analysis or image processing. Quantization reduces the number of        colors in an image. In the following example a K-Means        Clustering algorithm is used.

Using the features or metrics described above, a number of scores can beassigned to a student's work, and automatic feedback responses can begenerated. In this section we present selected examples of a fewautomatically gradable metrics:

Color Theory: Color quantization and color histograms can be used todetermine the dominant colors present in the image or frame, which canbe tested to see if they adhere to the basic principles of color theory,i.e., color harmony, complimentary, warm vs. cool, etc. Color theory isat the core of many visual based mediums, and as such, this type ofcolor analysis is extremely useful in the context of photography, video,painting, graphic design, and other visual arts.

Mood: Different “moods” can be linked to (or mapped, in the featureextraction, classification and mapping sense of our automated courseworkevaluation techniques, from) different hues, contrast, and saturation.In assignments where students are asked to recreate the look-and-feel ofreference works (for example, certain periods in time/history, darker,warm vs. cool), computationally evaluating the image histogram, averagehue, average contrast, and average saturation, can yield a score/gradein relation to how closely the student matched the look and feel of thereference source or “style” (e.g., film noir).

Color Correction. The technique of color correction is an importantskill in digital photography and video editing and, as a result, is anexpressive aspect that we seek to evaluate in coursework submissionsusing computational techniques described herein. In an assignment, astudent could be given an uncorrected image and told to match acorrected version to the best of their ability. Their courseworksubmission can then be graded against the reference image by comparingthe color histograms, overall contrast, and saturation values. A scoreor grade is derived based on how close their corrections matched thereference image and feedback.

Grading of Music Programming:

A related, and also illustrative, example of a media-rich educationaldomain in which techniques of the present invention may be employed ismusic programming, i.e., digital signal processing software as appliedto audio encodings of music. In a music programming course, students maybe given an assignment to develop a computer program to perform somedesired form of audio processing on an audio signal encoding. Forexample, a student might be assigned the task of developing programmingto reduce the dynamic range of an existing audio track (calledcompression) by computing a running average of the RMS power in thesignal, then applying a dynamically varying gain to the signal in orderto make louder segments softer, and softer segments louder, thuslimiting the total dynamic range.

In support of such an assignment, the systems and methods describedherein may perform an initial textual and structural evaluation of thestudents' submitted coursework (here, computer code). Using lexicaland/or syntactic analysis, it is possible to determine conformance withvarious elements required by the assignment, e.g., use of callingstructures and required interfaces, use of particular computationalprimitives or techniques per the assignment, coding within storage useconstraints, etc. Next, the systems and methods may compile thesubmitted code automatically (e.g., to see if it compiles; if not, thestudent must re-submit). Once compiled, the coursework submission may beexecuted against a data set to process audio input and/or generate audiooutput. Alternatively or in addition, the student's submission itselfmay include results (e.g., an encoded audio signal) generated byexecution of the coursework submission against a data set to processaudio input and/or generate audio output. In either case, the audiofeatures are extracted from the audio signal output and supplied to theclassifiers of the machine-grading system to produce a grading or scorefor the coursework submission. See e.g., FIG. 3 and the optional dataset 341 (for evaluation of submitted code) and optional compile andexecuted operations 342 illustrated therein.

Predefined Feature Extractors and Exemplary Classifier Designs:

Although the selection of particular audio features to extracted may be,in general, assignment- or implementation-dependent (and in some casesor embodiments may be augmented or extended with instructor- orcurriculum-defined feature extraction modules), an exemplary set ofaudio feature extraction modules may be provided for selection by theinstructor or curriculum designer. For example, in some cases orembodiments, the following computationally-defined feature extractionsmay be provided or selected with computations over windows of varioussize (20, 50, 100 ms, 0.5 s, typical):

-   -   RMS (Root Mean Square) energy of the audio signal;    -   Number of zero crossings (per frame) in the audio signal;    -   Spectral flux (frame to frame difference of power spectra, e.g.,        FFT magnitude) of the audio signal;    -   Spectral centroid (center of gravity of power spectrum,        brightness measure) of the audio signal;    -   Spectral roll-off frequency for the audio signal (below this        freq., X % of total power spectrum energy lies);    -   Spectral tilt of the audio signal (slope of line fit to power        spectrum or log power spectrum);    -   mel-frequency cepstral coefficients (MFCC) representation of        short-term power spectrum for the audio signal (inverse        transform of log of power spectrum, warped to Mel freq. scale);    -   Beat histogram for the audio signal (non-linear        autocorrelation-based estimates of music/sonic pulse); and/or    -   Multi-pitch histograms for the audio signal (extract sinusoids,        cluster by harmonicity, calculate pitches).

In general, means and standard deviations of these and/or or otherextracted features are computed (often over different windows) and usedto characterize sound and music. In some cases, signals may besegmented, and features computationally extracted overcontextually-specific segments. Using various metrics, distancefunctions, and systems, including artificial neural network (NN),k-nearest neighbor (KNN), Gaussian mixture model (GMM), support vectormachine (SVN) and/or other statistical classification techniques,sounds/songs/segments can be compared to others from a previously scoredor graded database of training examples. By classifying courseworksubmissions against a computational representation of the scored orgraded training examples, individual coursework submissions are assigneda grade or score. In some case or embodiments, features (or featuresets) can be compressed to yield “fingerprints,” makingsearch/comparison faster and more efficient. Based on the descriptionherein, persons of ordinary skill in the art will appreciate both a widevariety of computationally-defined features that may be extracted fromthe audio signal of, or derived from, a coursework submission and a widevariety of computational techniques for classifying such courseworksubmissions based on the extracted features.

Turning illustratively to visual art, images and/or video, it will beappreciated that it is possible to compute features from still images,or from a succession of images (video) using similar or at leastanalogous techniques applied generally as described above. Techniques inthis sub-domain of signal processing are commonly referred to as“computer vision” and, as will be appreciated by persons of ordinaryskill in these arts having benefit of the present disclosure, analogousfeatures for extraction can include color histograms, 2D Transforms,edges/corners/ridges, curves and curvature, blobs, centroids,optical-flow, etc. In video targeted applications, detection of sectionsand transitions (fades, cuts, dissolves, etc.) may be used in automatedgrading, particularly where a rubric asks students to use at least oneeach of jump-cut, fade, cross-dissolve, or to have at least threeseparate “scenes”. As with the audio processing examples above, decisiontree logic and computationally-defined features may be employed todetect sections and transitions (here, fades, cuts, dissolves, etc.)between sections. If statistics for sections differ, grading/scoring canbe based on the presence, character or structure of the sections ortransitions and correspondence with the rubric.

As before, means and standard deviations of these and/or or otherextracted features are computed (often over different windows) and usedto characterize the images or video. Again, extracted feature sets maybe used to classify coursework submissions against a graded or scoredset of exemplars. Using various metrics, distance functions, andsystems, including artificial neural network (NN), k-nearest neighbor(KNN), Gaussian mixture model (GMM), support vector machine (SVN) and/orother statistical classification techniques, images/video segments canbe compared to others from a previously scored or graded database oftraining examples. By classifying coursework submissions against acomputational representation of the scored or graded training examples,individual coursework submissions are assigned a grade or score.

Other Embodiments and Variations

While the invention(s) is (are) described with reference to variousembodiments, it will be understood that these embodiments areillustrative and that the scope of the invention(s) is not limited tothem. Many variations, modifications, additions, and improvements arepossible. For example, while certain illustrative signal processing andmachine learning techniques have been described in the context ofcertain illustrative media-rich coursework and curricula, persons ofordinary skill in the art having benefit of the present disclosure willrecognize that it is straightforward to modify the described techniquesto accommodate other signal processing and machine learning techniques,other forms of media-rich coursework and/or other curricula.

Both instructor-side and student-side portions of a feature extractionand machine learning system process flows for media-rich coursework havebeen described herein in accordance with some embodiments of the presentinvention(s). In simplified, yet illustrative use cases chosen toprovide descriptive context, the instructor or curriculum designerprovides a set of exemplars that she scores, classifies or labels as“good” and provides set of exemplars that she scores, classifies orlabels as “bad.” The instructor or curriculum, then trains theillustrated computational machine by selecting/pairing features orexpressing rules or other decision logic, as needed, to computationallyclassify the exemplars (and subsequent coursework submissions) in accordwith the desired classifications. As will be appreciated by persons ofordinary skill in the art having benefit of the present disclosure,scores, classes or labels of interest may be multi-level, multi-variate,and/or include less crass or facially apparent categorizations. Forexample, classifiers may be trained to classify in accordance withinstructor or curriculum provided scores (e.g., ratings from 0 to 6 oneach of several factors, on a 100-point scale or, in some cases, ascomposite letter grades) or labels (e.g., expert/intermediate/amateur),etc.

Furthermore, as online courses become more popular and are offered forcredit, a further concern arises related to verification andauthentication of the student taking the course and submittingassignments. Cases of fraud, e.g., where someone is hired to do the workfor someone else who will receive credit, must be avoided if possible.Some institutions who offer online for credit require physicalattendance at proctored examinations. As online courses expand to offercredit in geographically diverse locations, and as class sizes grow,supervised exams can become impractical or impossible. The techniquesimplemented by systems described herein can help with this problem,using the same or similar underlying frameworks for voice, face, andgesture recognition. If required, the user can be required toauthenticate themself, e.g., via Webcam, with each assignmentsubmission, or a number of times throughout an online exam. In somecases or embodiments, a student authentication may use the same orsimilar features used to grade assignments to help determine or confirmidentity of the coursework submitter For example, in some cases orembodiments, computationally-defined features extracted from audioand/or video provided in response to a “Say your name into themicrophone” direction or a “Turn on your webcam, place your face in thebox, and say your name” requirement may be sufficient to reliablyestablish (or confirm) identity of an individual taking a final exambased, at least in part, on data from earlier coursework submissions orenrollment.

Embodiments in accordance with the present invention(s) may take theform of, and/or be provided as, a computer program product encoded in amachine-readable medium as instruction sequences and other functionalconstructs of software, which may in turn be executed in a computationalsystem to perform methods described herein. In general, a machinereadable medium can include tangible articles that encode information ina form (e.g., as applications, source or object code, functionallydescriptive information, etc.) readable by a machine (e.g., a computer,server, virtualized compute platform or computational facilities of amobile device or portable computing device, etc.) as well asnon-transitory storage incident to transmission of the information. Amachine-readable medium may include, but is not limited to, magneticstorage medium (e.g., disks and/or tape storage); optical storage medium(e.g., CD-ROM, DVD, etc.); magneto-optical storage medium; read onlymemory (ROM); random access memory (RAM); erasable programmable memory(e.g., EPROM and EEPROM); flash memory; or other types of mediumsuitable for storing electronic instructions, operation sequences,functionally descriptive information encodings, etc.

In general, plural instances may be provided for components, operationsor structures described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of the invention(s). Ingeneral, structures and functionality presented as separate componentsin the exemplary configurations may be implemented as a combinedstructure or component. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents. These and other variations, modifications, additions, andimprovements may fall within the scope of the invention(s).

What is claimed is:
 1. A method for use in connection with automatedevaluation of coursework submissions, the method comprising: receivingfrom an instructor or curriculum designer a selection of exemplary mediacontent to be used in evaluating the coursework submissions, theexemplary media content including a training set of examples eachassigned at least one quality score by the instructor or curriculumdesigner; accessing computer readable encodings of the exemplary mediacontent that together constitute the training set and extracting fromeach instance of exemplary media content a first set of computationallydefined features; for each instance of exemplary media content,supplying a classifier with both the instructor or curriculum designer'sassigned quality score and values for the computationally definedfeatures extracted therefrom; based on the supplied quality scores andextracted feature values, training the classifier, wherein the trainingincludes updating internal states thereof; accessing a computer readableencoding of media content that constitutes, or is derived from, thecoursework submission, wherein the computer readable encoding of mediacontent includes an image or video frame processed or rendered bysoftware code included in the coursework submission; segmenting theimage or video frame to define at least one region of interest;extracting, from the computer readable encoding of media content, asecond set of computationally defined features, wherein a particularcomputationally defined feature of the second set is extracted from thesegmented at least one region of interest; and applying the trainedclassifier to the extracted second set of computationally definedfeatures and, based thereon, assigning a particular quality score to thecoursework submission.
 2. A method as in claim 1, further comprising:supplying plural additional classifiers with respective instructor orcurriculum designer's assigned quality scores and values forcomputationally defined features extracted from respective instances ofthe exemplary media content; training the additional classifiers; andapplying the trained additional classifiers.
 3. A method as in claim 1,wherein the quality score is, or is a component of, a grading scale foran assignment- or test question-type coursework submission.
 4. A methodas in claim 1, wherein the particular computationally defined featurecharacterizes one of: size or position of the region of interestrelative to a rule of thirds alignment or point; and focus of, orwithin, the region of interest relative at least one other portion ofthe image or video frame.
 5. A method as in claim 1, wherein thesegmenting provides foreground/background segmentation.
 6. A method asin claim 1, wherein the segmenting includes adaptively refined keypointsets, wherein adaptive refinement is based on at least one of: iterationon the keypoint sets using automatically updated keypoint detectionparameters; and computational clustering to refine the keypoint sets. 7.A method as in claim 1, wherein the software code is submitted insatisfaction of a programming assignment or test question, the softwarecode executable to perform, or compilable to execute and perform,digital signal processing to produce output media content; wherein theexemplary media content includes exemplary output media content producedusing exemplary software codes; and wherein the particular quality scoreassigned to the coursework submission is based on the applying of theclassifier to the second set of computationally defined featuresextracted from the output media content produced by execution of thesubmitted software code.
 8. A method as in claim 7, wherein the softwarecode coursework submission is executable to perform digital signalprocessing on input media content to produce the output media content;and wherein the exemplary output media content is produced from theinput media content using the exemplary software codes.
 9. A method asin claim 1, wherein the coursework submission includes a computerreadable media encoding of expressive media content selected from theset of: sketches, paintings, photographic images or other artistic stillvisuals; and synchronized audiovisual content, computer animation orother video that is itself expressive or visually captures underlyingexpression such as dance, acting, or other performance.
 10. A method asin claim 1, wherein the media content that constitutes, or is derivedfrom, the coursework submission includes an audio signal encoding; andwherein for the first and second sets, at least some of thecomputationally defined features are selected or derived from: a rootmean square energy value; a number of zero crossings per frame; aspectral flux; a spectral centroid; a spectral roll-off measure; aspectral tilt; a mel-frequency cepstral coefficients (MFCC)representation of short-term power spectrum; a beat histogram; and/or amulti-pitch histogram computed over at least a portion of the audiosignal encoding.
 11. A method as in claim 1, wherein for the first andsecond sets, at least some of the computationally defined features areselected or derived from: color histograms; two-dimensional transforms;edge, corner or ridge detections; curve or curvature features; a visualcentroid; and/or optical flow computed over at least a portion of theimage or video frame.
 12. A method as in claim 1, wherein the classifierimplements an artificial neural network (NN), k-nearest neighbor (KNN),Gaussian mixture model (GMM), support vector machine (SVN) or otherstatistical classification technique.
 13. A method as in claim 1,further comprising: iteratively refining the classifier training basedon supply of successive instances of the exemplary media content to theclassifier and updates to internal states thereof.
 14. A method as inclaim 13, further comprising: continuing the iterative refining until anerror metric based on a current state of classifier training falls belowa predetermined or instructor or curriculum designer-defined threshold.15. A method as in claim 1, wherein the classifier is implemented usingone or more logical binary decision trees, blackboard voting-typemethods, or rule-based classification techniques.
 16. A method as inclaim 1, further comprising: supplying the instructor or curriculumdesigner with an error metric based on a current state of classifiertraining.
 17. A method as in claim 1, further comprising: supplying theinstructor or curriculum designer with a coursework task recommendationbased on a particular one or more of the computationally definedfeatures that contribute most significantly to classifier performanceagainst the training set of exemplary media content.
 18. A method as inclaim 1, wherein the first and second sets of computationally definedfeatures are the same.
 19. A method as in claim 1, wherein the secondset of computationally defined features includes a subset of the firstset features selected based on contribution to classifier performanceagainst the training set of exemplary media content.
 20. A method as inclaim 1, wherein the quality score is, or is a component of a gradingscale for an assignment- or test question-type coursework submission.21. A method as in claim 1, further comprising: receiving from theinstructor or curriculum designer at least an initial definition of thefirst set of computationally defined features.
 22. A computationalsystem including one or more operative computers programmed to performthe method of claim
 1. 23. The computational system of claim 22embodied, at least in part, as a network deployed coursework submissionsystem, whereby a large and scalable plurality (>50) of geographicallydispersed students may individually submit their respective courseworksubmissions in the form of computer readable information encodings. 24.The computational system of claim 23 including a student authenticationinterface for associating a particular coursework submission with aparticular one of the geographically dispersed students.
 25. Anon-transient computer readable encoding of instructions executable onone or more operative computers to perform the method of claim
 1. 26. Acoursework management system for automated evaluation of courseworksubmissions, the system comprising: an instructor or curriculum designerinterface for selecting or receiving exemplary media content to be usedin evaluating the coursework submissions, the exemplary media contentincluding a training set of examples each assigned at least one qualityscore by the instructor or curriculum designer; a training subsystemcoupled and programmed to access computer readable encodings of theexemplary media content that together constitute the training set and toextracting from each instance of exemplary media content a first set ofcomputationally defined features; the training subsystem furtherprogrammed to, for each instance of exemplary media content, supply aclassifier with both the instructor or curriculum designer's assignedquality score and values for the computationally defined featuresextracted therefrom, and to, based on the supplied quality scores andextracted feature values, train the classifier, wherein the trainingincludes updating internal states thereof; and a coursework evaluationdeployment of the trained classifier coupled and programmed to access acomputer readable encoding of media content that constitutes, or isderived from, the coursework submissions and to extract therefrom asecond set of computationally defined features, wherein the mediacontent that constitutes, or is derived from, the coursework submissionincludes an image or video frame; wherein the coursework evaluationdeployment is configured to: segment the image or video frame to defineat least one region of interest; extract, for the segmented at least oneregion of interest, a particular computationally defined feature of thesecond set; and apply the trained classifier to the extracted second setof computationally defined features and, based thereon, assigns aparticular quality score to the coursework submission.
 27. Thecoursework management system as in claim 26, wherein the trainingsubsystem supplies plural additional classifiers with respectiveinstructor or curriculum designer's assigned quality scores and valuesfor computationally defined features extracted from respective instancesof the exemplary media content and trains the additional classifiers;and wherein coursework evaluation deployment also applies the trainedadditional classifiers.
 28. The coursework management system as in claim26, further comprising: an execution environment, wherein the courseworksubmission includes software code submitted in satisfaction of aprogramming assignment or test question, the software code executable inthe execution environment to perform, or compilable to execute in theexecution environment and perform, digital signal processing to produceoutput media content; wherein the exemplary media content includes theoutput media content produced using the submitted software code; andwherein the particular quality score assigned to the courseworksubmission is based on the applying of the classifier to the second setof computationally defined features extracted from the output mediacontent produced using the submitted software code.
 29. The courseworkmanagement system as in claim 28, wherein the output media contentincludes audio signals processed or rendered by the software codecoursework submission.
 30. The coursework management system as in claim26, wherein the classifier implements an artificial neural network (NN),k-nearest neighbor (KNN), Gaussian mixture model (GMM), support vectormachine (SVN) or other statistical classification technique.
 31. Thecoursework management system as in claim 26, wherein the trainingsubsystem allows the instructor or curriculum designer to iterativelyrefine the classifier training based on supply of successive instancesof the exemplary media content to the classifier and updates to internalstates thereof.
 32. The coursework management system as in claim 26,wherein the classifier is implemented using one or more logical binarydecision trees, blackboard voting-type methods, or rule-basedclassification techniques.
 33. A coursework management systemcomprising: means for selecting or receiving exemplary media content tobe used in evaluating coursework submissions, the exemplary mediacontent including a training set of examples each assigned at least onequality score by an instructor or curriculum designer; means forextracting from each instance of exemplary media content a first set ofcomputationally defined features, for supplying a classifier with boththe instructor or curriculum designer's assigned quality score andvalues for the computationally defined features extracted therefrom and,based on the supplied quality scores and extracted feature values, fortraining the classifier; means for accessing a computer readableencoding of media content that constitutes, or is derived from, thecoursework submission and extracting therefrom a second set ofcomputationally defined features, wherein the media content thatconstitutes, or is derived from, the coursework submission includes animage or video signal encoding; means for segmenting the image or videoframe to define at least one region of interest; means for extractingfrom the coursework submissions a second set of computationally definedfeatures, including extracting, for the segmented at least one region ofinterest, a particular computationally defined feature of the secondset; and means for applying the trained classifier to the extractedsecond set of computationally defined features and, based thereon, forassigning a particular quality score to the coursework submission.